I’ve been experimenting with vSphere’s memory hot add and CPU hot plug features to determine its usefulness with Windows Server operating systems. I came up with mixed results depending on the version and architecture of the OS.
A few notes about the results:
- Memory hot remove is not supported at all by vSphere. It’s not an option no matter what the guest OS.
- Although virtual hardware can be hot added depending on the OS, there are caveats in certain cases
- A guest reboot may be required (this is outlined in the table below).
- Memory that is hot added to guests that support the hot add without a reboot will result in 100% sustained CPU utilization in the guest OS for a variable period of time that is dependent on the amount of of memory that is added. In my testing (and keep in mind your mileage may vary on different hardware):
- 1GB of RAM hot added resulted in 100% CPU for 1-3 seconds.
- 3GB of RAM hot added resulted in 100% CPU for about 10 seconds.
- CPU hot unplug is supported by vSphere but was not supported by any of the Windows operating systems that I tested.
- Going from 1vCPU to 2vCPUs in Windows 2008 guest operating systems did not result in a HAL change. From what I can tell, Windows 2008 uses the same HAL for uniprocessor and SMP. When a vCPU is hot added, it does show up right away in the Device Manager, however, it’s not seen in Task Manager or Computer Properties therefore my assumption is that processes are not being scheduled on the added vCPU until after the reboot at which time the additional vCPU shows up in all places that it should (ie. Task Manager, Computer Properties, etc.)
- I certainly like the innovation and flexibility here but I’m not sure hot add technology is going to mesh well with planned change management systems. The most important thing to recognize though is that VMware offers this technology to us as our choice to use or not use. It’s not a feature VMware held back drawing their own conclusion that nobody on the planet could ever use it. Microsoft does this today with Hyper-V memory over commit. Or rather they don’t offer memory over commit in Hyper-V because they made the decision on behalf of all their customers that nobody could or should use memory over commit. Instead you should pad your hosts with more physical memory at additional cost to you.
Here is the table of results I came up with:
| Memory hot add |
Memory hot remove |
CPU hot plug |
CPU hot unplug |
|
| Windows Server 2003 STD x86 | 🙁 | 🙁 | 🙁 | 🙁 |
| Windows Server 2003 STD x64 | 🙁 | 🙁 | 🙁 | 🙁 |
| Windows Server 2003 ENT x86 | 😎 | 🙁 | 🙁 | 🙁 |
| Windows Server 2003 ENT x64 | 😎 | 🙁 | 🙁 | 🙁 |
| Windows Server 2008 STD x86 | 😎 * | 🙁 | 🙁 | 🙁 |
| Windows Server 2008 STD x64 | 😎 * | 🙁 | 😎 * | 🙁 |
| Windows Server 2008 ENT x86 | 😎 | 🙁 | 🙁 | 🙁 |
| Windows Server 2008 ENT x64 | 😎 | 🙁 | 😎 * | 🙁 |
| Windows Server 2008 DC x86 | 😎 | 🙁 | 🙁 | 🙁 |
| Windows Server 2008 DC x64 | 😎 | 🙁 | 😎 | 🙁 |
| Windows Server 2008 R2 DC x64 (experimental support only) |
😎 | 🙁 | 😎 | 🙁 |
| * Reboot of guest OS required to recognize added hardware | ||||
Solaris supports such and has for years. You sure this isn’t a MS issue?
VMware vSphere provides the hot add and hot remove virtual hardware at the hypervisor layer. Caveats at the guest OS level are the responsibility of the guest OS.
kind of a blow off response.
When vSphere is downloadable I’ll test and post results dealing with more than a single view (Microsoft based guest systems).
I am guessing solaris will be fine. Some linux builds will be as well if base system is compiled for it. I’d bet the *BSD systems will fail much like your limited testing results with just MS systems did.
I don’t know much about Solaris – I’ve not had an opportunity to use it in my lifetime.
The first sentence in this blog post reads “I’ve been experimenting with vSphere’s memory and CPU hot plug (hot add) features to determine its usefulness with Windows Server operating systems.”
The hot add tests I performed with Windows were not designed to be all inclusive.
Hi Jason
been testing this too, on the Windows front only windows 2008 data centre edition supports hot add cpu, with the others they show in the device manager but a reboot is required – with data centre editon the pop right into the task manager
Andy is correct — this feature requires Datacenter x64, at least that is what I recall hearing at an internal vSphere demo a couple of weeks ago. I’m sure this is in the vSphere docs but I don’t have them handy.
Nice writeup; I am looking forward to your next “new in vSphere post.” Keep spreading the good news!
In my testing with hot add CPU/memory the VM behaved a little bit like it does during VMotion. That is, it dropped a ping or two but otherwise was ok. I didn’t experience the 10 second freezes but if it drops pings for a full 10 seconds then this option has limited usefulness at best for production workloads. Still, I agree that this is one of those features where we should be the ones deciding when it is appropriate to use it.
The behavior of hot-add CPU is exactly as described – it shows up in Device Manager but the OS doesn’t detect it until a reboot. I proved it by running CPU benchmarks before, during, and after and saw almost no utilization of the new CPU. After rebooting it used both CPUs.
It’ll be interesting to see where this goes. VMotion started out as a manual process and then DRS turned it into a fully automated solution. Maybe DRS will one day be extended to use this feature to add memory or processing power on the fly if needed…
Nice post Jason..
Matt
I would like to clarify that when I said CPU was at 100%, that didn’t necessarily mean the VM was frozen or unavailable on the network. I was not performing network tests or VM/application availability tests while hot adding memory.
Hey Jason,
What hardware are you using for your testing? I’m using a DL385 G1 and am only seeing a very short spike in CPU utilization during the hot adds. Just wondering if it is dependent on the hardware of the ESX host.
I ran more tests on the hot-add for memory. If you add memory during a memory test, the additional memory isn’t used. But if you simply stop the test and start it again it does start using the extra memory. If this were an Exchange or SQL server it would probably never see the memory since it is never starting a new instance of the process, so a reboot is probably required for this too. But I think this is more useful than the hot-add CPU.
Matt
Matt, I’m actually using the same host hardware you are – DL385 (G1) w/ a single DC AMD processor and 4 GB RAM. My vCenter server for vSphere is a VM running on faster processors in a different virtual infrastructure.
Check out Microsoft Dynamic Hardware Partitioning table for supported configurations.
http://msdn.microsoft.com/en-us/library/bb742871.aspx
None of the Microsoft OS’s support Hot-Remove of CPU or Memory.
Also, the key is to ensure you have the correct HAL on the server before attempting the hot-add CPU’s. I encountered this with the customers hot adding physical CPU’s to enterprise class servers at a past job. Regardless of physical or virtual, the Windows OS’s still needs to suspend processing for a brief period of time as new CPU or Memory resources are added to the OS with Windows.
Clarification:
Windows Server 2003 SP1 Enterprise Edition and Datacenter Edition support hot add of memory on x86-based, x64-based, and Itanium-based systems.
Windows Server 2008 supports hot add of processors, memory, and I/O host bridges plus hot replace of processors and memory on x64-based and Itanium-based systems.
Note: Although Windows Server 2008 does not support hot remove, Microsoft is considering including support for hot remove in a future version of Windows Server.
Source: Microsoft Windows Server 2008 Dynamic Hardware Partitioning White Paper
Long Version: http://download.microsoft.com/download/8/2/f/82fa3808-7168-46f1-a07b-f1a7c9cb4e85/WS08%20Dynamic%20Hardware%20Partitioning%20White%20Paper%20Final.doc
Short Version: http://tinyurl.com/ptrzmn
Thanks for the post, I like the amount of testing you did across multiple OS versions/architectures. Since Windows Server Enterprise Edition allows you do 4 virtual installations adding memory on the fly could be useful. However I would think in most situations (at least in small/medium businesses) that you would have already configured the guest for the appropriate amount of memory.
Jason,
You don’t happen to see a hot remove for virtual disks within a running VM in vsphere do you? Since it’s not in the marketing sheet I doubt it’s there but I thought I would ask.
cdarn: Yep, hot remove of virtual disk is in vSphere.
I think it may be different for each application.
I remember seeing a demo of CPU Hot plugging for Windows prior to 2008’s release, where an actual command had to be issued to inform Windows to start sharing the load across the newly added CPU.
SQL Server:
See here: http://technet.microsoft.com/en-us/library/bb964703.aspx
For some info about hot-adding CPUs in Windows.
“*SQL Server does not automatically start to use CPUs after they are added. This prevents SQL Server from using CPUs that might be added for some other purpose. After adding CPUs, execute the RECONFIGURE statement, so that SQL Server will recognize the new CPUs as available resources.”
So basically, I think reboot is a sure thing, or restart all your applications, but some apps should let you change their affinity live.
Even if that’s just SQL and Exchange, well, those are pretty common apps you see on Windows servers.
Great comments. I think it’s going to be quite a while before change management is ready for hot add cpu or memory. I know VMotion was a new concept for changement and many people like to make that comparison, but with hot add/remove, we’re making major changes to the guest OS. This really takes things to a whole new level.
Thanks for the good article. I appreciate your insights!
The article is looking very good i must say.
The only thing that is bothering me is the wrong info about X86 and X64
X86 is saying something abou the Intell architecture and doesn’t say anything about 32 or 64 bit. X86 cis and will remain the architecture layout.
IA64 was the 1st new architecture of Intell which had a BIT comment in the name: Itanium Archtecture 64 (bit)
are there any hot-add-remove of cpu and memory for linux OS? do you have a matrix?
Hi, All;
If you want check if your OS guest supports Hot-Plug in ESX (or other VMware products). Check at:
http://www.vmware.com/resources/compatibility/search.php?deviceCategory=software
For example: RHEL 6.1 supports Hot-Plug of vCPU and vMem (Hot Add Memory and Hot Add vCPU)
Thanks Man for the Post I was scouring the entire net to check if Hot Memory Reduction is possible inside a VM without reboot. I suspect it is if the OS is RHEL 5.6 on vSphere. It wold be great if you could check
You can hot add a CPU and the Windows OS will detect it, but any process that was running before the add will need to have the cpu(s) added to the process affinity table (right click the process and click affinity).
I love this table, any plans to update it?
Funny comment above from Mike.
According to VMware Solaris does NOT support either hotadd memory or hotplug CPU.